Papers with MRC models
Improving the Robustness of Deep Reading Comprehension Models by Leveraging Syntax Prior (D19-58)
Copied to clipboard
| Challenge: | Recent studies indicate that the current machine reading comprehension systems suffer from weak robustness against adversarial samples. |
| Approach: | They propose to take sentence syntax as the leverage in the answer predicting process and exploit the syntactic elements of a question to improve the generalization and robustness of MRC models. |
| Outcome: | The proposed method improves generalization and robustness against adversarial samples, with performance well-maintained. |
On the Robustness of Reading Comprehension Models to Entity Renaming (2022.naacl-main)
Copied to clipboard
| Challenge: | SpanBERT model is more robust than RoBERTa, despite having similar accuracy on unperturbed test data. |
| Approach: | They propose a pipeline to replace entity names with names from a variety of sources. |
| Outcome: | The proposed model performs worse when entities are renamed, the authors show . SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa . |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
Benchmarking Robustness of Machine Reading Comprehension Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate models' robustness under test-time perturbations or adversarial attacks. |
| Approach: | They propose a model-agnostic benchmark to evaluate models' robustness under adversarial attacks. |
| Outcome: | The proposed model-agnostic benchmark evaluates models under four different types of adversarial attacks. |
Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension (C18-1)
Copied to clipboard
| Challenge: | Existing MRC algorithms assume that each question is answerable by looking at text passages, but to realize human-like language comprehension ability, a machine should be able to distinguish not-answerable questions from answerable questions. |
| Approach: | They propose a method for automatically assigning difficulty level labels to a dataset that alters an existing MRC dataset and describes the resulting dataset. |
| Outcome: | The proposed method can detect NAQs in a dataset with difficulty level labels and is valid and potentially useful in the development of advanced MRC models. |
Why Machine Reading Comprehension Models Learn Shortcuts? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that many MRC models learn shortcuts to outwit benchmarks, but the performance is unsatisfactory in real-world applications. |
| Approach: | They propose to use shortcut questions to analyze learning difficulty of MRC models . they propose to analyze the learning difficulty regarding shortcut and challenging questions . |
| Outcome: | The proposed methods show that a large proportion of shortcut questions in training data make models rely on shortcut tricks excessively. |
To Answer or Not To Answer? Improving Machine Reading Comprehension Model with Span-based Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models fail to recognize answerable questions due to subtle literal changes . MRC models are forced to perceive crucial semantic changes from slight literal differences. |
| Approach: | They propose a span-based method of Contrastive Learning which explicitly contrasts answerable questions with their answerable counterparts at the answer span level. |
| Outcome: | The proposed method improves baselines significantly and is an effective way to utilize generated questions. |
RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) achieve high recall amongst top few predictions, but low overall accuracy, motivating the need for answer re-ranking. |
| Approach: | They propose a method to make answer re-ranking successful for span-extraction tasks even beyond large pre-training. |
| Outcome: | The proposed approach achieves 45.5% Exact Match accuracy on Natural Questions and 61.7% on TriviaQA. |
The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. |
| Approach: | They propose to fine-tune three state-of-the-art language models on SQuAD 1.1 or SQu AD 2.0 and then evaluate their robustness under adversarial attacks. |
| Outcome: | The proposed model is able to perform better under adversarial attacks than model fine-tuned on SQuAD 1.1 or 2.0. |
DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications (2021.acl-short)
Copied to clipboard
| Challenge: | In order to comprehensively verify the robustness and generalization of MRC models, we construct a real-world Chinese dataset - DuReader_robust . |
| Approach: | They introduce a real-world Chinese dataset to evaluate the robustness and generalization of MRC models from three aspects: over-sensitivity, over-stability and generalisation. |
| Outcome: | The proposed model fails to perform well on the challenge test set and may provide suggestions for future model development. |
Learning to Generate Questions by Learning to Recover Answer-containing Sentences (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent research has focused on synthetically generating a question from a given context and an annotated answer by training an additional generative model. |
| Approach: | They propose a method that learns to generate contextually rich questions by recovering answer-containing sentences. |
| Outcome: | The proposed approach improves the quality and accuracy of existing models and achieves comparable results to the state-of-the-art on MS MARCO and NewsQA. |
Machine Reading Comprehension as Data Augmentation: A Case Study on Implicit Event Argument Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets are too small to train a model for capturing regularities underlying how event arguments are extracted. |
| Approach: | They propose to bridge implicit EAE with machine reading comprehension (MRC) by building a unified training framework and explicit data augmentation regimes via MRC. |
| Outcome: | The proposed method obtains state-of-the-art performance on two benchmarks and demonstrates superior results in a data-low scenario. |
Explicit Utilization of General Knowledge in Machine Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Existing MRC models are unable to integrate general knowledge with human knowledge. |
| Approach: | They propose a data enrichment method which uses WordNet to extract inter-word semantic connections as general knowledge from each given passage-question pair. |
| Outcome: | The proposed model outperforms state-of-the-art models and is robust to noise. |
A Vietnamese Dataset for Evaluating Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | despite the lack of benchmark datasets for Vietnamese, there are few studies on machine reading comprehension (MRC) . MRC is an essential core for a range of natural language processing applications such as search engines and intelligent agents. |
| Approach: | They propose to use Vietnamese Question Answering Dataset to evaluate machine reading comprehension in Vietnamese . they use over 23,000 human-generated question-answer pairs based on 5,109 Vietnamese articles . |
| Outcome: | The proposed dataset includes over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Vietnamese articles from Wikipedia. |
Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure (2020.coling-main)
Copied to clipboard
| Challenge: | Multiparty dialog applications such as discourse parsing and meeting summarization are now mainstream research. |
| Approach: | They propose to annotate a machine reading comprehension dataset with discourse structure built over multiparty dialog using a modified Segmented Discourse Representation Theory (SDRT) style. |
| Outcome: | The proposed dataset contributes large-scale discourse dependency annotations in a modified Segmented Discourse Representation Theory (SDRT) style for all of its multiparty dialogs, and achieves only 67.7% F1 on Molweni’s questions, a 20+% significant drop as compared against its SQuAD 2.0 performance. |
Robust Machine Reading Comprehension by Learning Soft labels (2020.coling-main)
Copied to clipboard
| Challenge: | Neural models have achieved great success on the task of machine reading comprehension, which are typically trained on hard labels. |
| Approach: | They propose a robust training method for machine reading comprehension models to address label sparseness problem by using three strategies to train models on soft labels. |
| Outcome: | The proposed method improves the baseline model performance and achieves state-of-the-art performance on NewsQA and QUOREF. |
Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data. |
| Approach: | They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. |
| Outcome: | The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning. |
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension (N19-1)
Copied to clipboard
| Challenge: | Existing models for Machine Reading Comprehension (MRC) are small, compared to their size, and there are many studies on using pre-trained word embeddings and back-translation approaches to improve model generalization. |
| Approach: | They propose a multi-task learning framework to learn a machine reading comprehension model that can be applied to a wide range of MRC tasks in different domains. |
| Outcome: | The proposed model can be applied to a wide range of MRC tasks in different domains. |
A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension lack evidence labels for training models. |
| Approach: | They propose a method which supervises the evidence extractor with auto-generated evidence labels in an iterative process. |
| Outcome: | The proposed method improves on three MRC tasks on seven datasets. |
Coreference Reasoning in Machine Reading Comprehension (2021.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for machine reading comprehension do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Approach: | They propose to use existing coreference resolution datasets to train machine reading comprehension models to better reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Outcome: | The proposed method improves the performance of state-of-the-art models on a set of coreference-related datasets. |
Contrastive Distant Supervision for Debiased and Denoised Machine Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Distant supervision (DS) is a promising learning approach for machine reading comprehension (MRC) however, the annotated dataset will inevitably lead to mislabeled instances, resulting in answer bias and context noise problems. |
| Approach: | They propose an algorithm that can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. |
| Outcome: | The proposed algorithm can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. |
IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing MRC datasets in Indonesian are inadequate because of the small size and limited question types. |
| Approach: | They propose to combine automatic and manual unanswerable question generation to minimize the cost of manual dataset construction while maintaining the dataset quality. |
| Outcome: | The proposed dataset significantly improves the performance of Indonesian MRC models, showing a large improvement for unanswerable questions. |
Natural Response Generation for Chinese Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | MRC models trained on labeled answers are limited in generating human-like responses in real QA scenarios. |
| Approach: | They construct a dataset called Penguin to promote machine reading comprehension . they use 200k training data with fluent, well-informed responses to train models . |
| Outcome: | The proposed dataset is the first benchmark towards natural response generation in Chinese MRC on a relatively large scale. |
Causal Intervention for Mitigating Name Bias in Machine Reading Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing MRC models may overuse name information to make predictions, causing name bias . |
| Approach: | They propose a Causal Interventional paradigm for MRC to mitigate name bias by analyzing pre-trained knowledge and context representations. |
| Outcome: | The proposed model is robust to names and performs competitively on the original SQuAD. |